Chapter 15: Second Year Evaluation of the ADAPT Program

G i v e n t h a t t h e f i r s t y e a r ' s e v a l u a t i o n i n d i c a t e d t h a t many of t h e g o a l s were b e i n g a t t a i n e d , it was d e c i d e d t h a t d u r i n g t h e s e c o n d y e a r we s h o u l d u s e more s t r i n g e n t m e a s u r e s o f . s u c c e s s , T h e m e a s u r e m e n t o f c h a n g e i n a b s t r a c t o r f o r m a l o p e r a t i o n a l t h i n k i n g s k i l l s p r o v i d e s a n e x a m p l e , D u r i n g t h e f i r s t y e a r a number o f t a s k s were u s e d w h i c h h a d b e e n d e v e l o p e d f o r r e s e a r c h p u r p o s e s . A l t h o u g h t h e s e t a s k s were t h e o r e t i c a l l y g e a r e d t o a s s e s s f o r m a l o p e r a t i o n a l t h o u g h t , t h e y d i d not meet strict test c o n s t r u c t i o n r e q u i r e m e n t s . It seemed t h a t o t h e r n a t i o n a l l y s t a n d a r d i z e d tests o f c r i t i c a l t h i n k i n g m i g h t d e m o n s t r a t e more c o n v i n c i n q l y t h e v a l u e of t h e ADAPT p r o g r a m i n f o s t e r i n g t h i n k i n g s k i l l s . Hence , a n a t i o n a l l y normed, o b j e c t i v e test -t h e Watson G l a s e r C r i t i c a l T h i n k i n g A p p r a i s a l -was a d m i n i s t e r e d d u r i n g t h e s e c o n d y e a r . A s imilar c h a n g e w i t h r e s p e c t t o t h e m e a s u r e of p e r s o n a l i t y d e v e l o p m e n t o c c u r r e d , D u r i n g t h e f i rs t y e a r a t e s t of c o n c e p t u a l c o m p l e x i t y was u s e d . T h i s test , a l t h o u g h u s e d w i d e l y t o a s s e s s i n t e r p e r s o n a l d e v e l o p m e n t , d o e s n o t meet t h e p a r a m e t r i c r e q u i r e m e n t s t y p i c a l l y demanded of n a t i o n a l l y normed a s s e s s m e n t s . Here a g a i n , it was d e c i d e d t h a t ACAPT's s u c c e s s c o u l d b e d e m o n s t r a t e d c o n v i n c i n g l y i f a more r i g o r o u s a s s e s s m e n t was u s e d . H e n c e , J a n e L o e v i n g e r q s m e a s u r e o f Ego D e v e l o p m e n t became a p a r t o f t h e s e c o n d y e a r e v a l u a t i o n . The t h o u g h t s of t h e e v a l u a t i o n team a n d p r o j e c t s t a f f c o n c e r n i n q t h e s e c o n d y e a r e v a l u a t i o n are r e f l e c t e d i n t h e s e c h a n g e s i n m e a s u r e s . I n e s s e n c e , it seemed t h a t t h e p r o g r a m s u c c e s s f u l l y met its g o a l s d u r i n g t h e f i r s t y e a t o f its o p e r a t i o n . I n o r d e r t o a r g u e f o r w i d e r d i s s e m i n a t i o n of t h e , p r i n c i p l e s of t h e ADAPT p r o g r a m , h o w e v e r , we f e l t c o m p e l l e d t o p u t t h e p r o g r a m to a n o s e r i g o r o u s e v a l u a t i o n . T h e r e s u l t s of t h i s e v a l u a t i o n a r e r e p o r t e d i n s u b s e q u e n t s e c t i o n s . DESIGN OF THE EVALUATION T h e e v a l u a t i o n i n s t r u m e n t s u s e d , t h e d i f f e r e n t g r o u p s of s u b j e c t s i n v o l v e d i n t h e a s s e s s m e n t s , a n d t h e times when v a r i o u s tests were a d m i n i s t e r e d a r e d e p i c t e d i n T a b l e 1. I n s u b s e q u e n t s e c t i o n s t h e tests a n d s u b j e c t s a m p l e s a re d e s c r i b e d a n d t h e r a t i o n a l e f o r t h e d e s i g n is g i v e n . 1. W a t s o n G l a s e r C r i t i c a l T h i n k i n g A p p r a i s a l . T h i s test c o n s i s t s of a series of e x e r c i s e s w h i c h r e q u i r e t h e a p p l i c a t i o n o f s o m e o f t h e i m p o r t a n t a b i l i t i e s i n v o l v e d i n c r i t i c a l t h i n k i n g : i n f e r e n c e ; r e c o g n i t i o n of a s s u m p t i o n s ; d e d u c t i o n ; i n t e r p r e t a t i o n ; a n d e v a l u a t i o n of a r g u m e n t s . I n a d d i t i o n , t h e c r i t i ca l t h i n k i n g a p p r a i s a l c a l & s f o r r e s p o n s e s t o t w o d i f f e r e n t k i n d s of items. I n some items t h e s t u d e n t c o n f r o n t s p r o b l e m s i n v o l v i n g n e u t r a l t o p i c s a b o u t w h i c h p e o p l e g e n e r a l l y d o n o t h a v e s t r o n g f e e l i n g s . O t h e r items, a p p r o x i m a t e l y p a r a l l e l i n l o g i c a l s t r u c t u r e , p e r t a i n t o p o l i t i c a l , e c o n o m i c , a n d soc i a l issues w h i c h are more l i k e l y t o a r o u s e e m o t i o n a l f e e l i n g s , b i a s e s , o r p r e j u d i c e s (Wat son G Glaser, 1 9 6 4 ) . T h i s t e s t h a s b e e n a d m i n i s t e r e d t o 5 , 6 7 5 f r e s h m e n i n 16 l i ~ e r a l a r t s c o l l e g e s . Hence , n o r m s f o r t h e p e r f o r m a n c e o f f r e s h m e n a r e a v a i l a b l e . I n a d d i t i o n , r e l i a b i l i t y a n d v a l i d i t y d a t a h a v e b e e n c o m p i l e d . F i n a l l y , t h e W a t s o n G l a s e r is a v a i l a b l e i n m u l t i p l e f o r m s . U s i n q two forms f o r p r e a n d p o s t t e s t i n g h a s t h e a d v a n t a g e of m i n i m i z i n g a n y p r a c t i c e effects. I n e s s e n c e , t h e u s e of t h e Y a t s o n G l a s e r C r i t i c a l T h i n k i n g A p p r a i s a l allows f o r a r i q o r o u s test o f w h e t h e r a n y c h a n g e s h a v e o c c u r r e d i n t h i n k i n g s k i l l s a s a r e s u l t o f t h e ADAPT Frogram. It is h y p o t h e s i z e d t h a t t h e ADAPT s t u d e n t s w i l l show s i g n i f i c a n t l y greater i n c r e a s e s o n t h e W a t s o n G l a s e r C r i t i c a l T h i n k i n g